Skip to content

🎨🐛 Fix filtering bug in celery tasks #8355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 40 commits into from
Sep 12, 2025
Merged

🎨🐛 Fix filtering bug in celery tasks #8355

merged 40 commits into from
Sep 12, 2025

Conversation

@bisgaard-itis
Copy link
Contributor

@bisgaard-itis bisgaard-itis commented Sep 11, 2025 •
edited
Loading

What do these changes do?

  • Fix a critical bug in the TaskFiltering class which meant that all users had access to all celery tasks anywhere 😱. This is because extra fields were ignored in that model. This PR changes that and allows them.
  • I took the chance to kill another problem we had since some time: This PR introduces a flexible way of filtering tasks when listing them (see Evolve TaskContext data structure to a TaskFilter #8071). The functionality for doing so is all done in the TaskFilter class.
  • Furthermore I added validators in TaskFilters to check that inputs are compatible with the assumptions we require to build Redis keys.
  • Added a bunch of tests to test the above.
  • Along the way I discovered some wrong usage of the client_name variable in AsyncJobFilter, so I corrected that (which is crucial for listing)

Related issue/s

How to test

  • Run tests in packages/service-library/tests/test_celery.py
  • Run tests in packages/celery-library/tests/unit/test_tasks.py

Dev-ops

@bisgaard-itis bisgaard-itis self-assigned this Sep 11, 2025
@bisgaard-itis bisgaard-itis added this to the Cheops milestone Sep 11, 2025
@codecov
Copy link

codecov bot commented Sep 11, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 93.25843% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.88%. Comparing base (ae14233) to head (47ab209).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #8355      +/-   ##
==========================================
+ Coverage   87.83%   87.88%   +0.05%     
==========================================
  Files        1945     1944       -1     
  Lines       75582    75603      +21     
  Branches     1314     1316       +2     
==========================================
+ Hits        66389    66447      +58     
+ Misses       8796     8759      -37     
  Partials      397      397
Flag Coverage Δ
integrationtests 63.96% <90.90%> (-0.10%) ⬇️
unittests 86.56% <93.25%> (+0.06%) ⬆️
Components Coverage Δ
pkg_aws_library 93.59% <ø> (ø)
pkg_celery_library 84.71% <100.00%> (-0.50%) ⬇️
pkg_dask_task_models_library 79.33% <ø> (ø)
pkg_models_library 93.10% <100.00%> (ø)
pkg_notifications_library 85.20% <ø> (ø)
pkg_postgres_database 88.02% <ø> (ø)
pkg_service_integration 70.19% <ø> (ø)
pkg_service_library 71.95% <87.80%> (+0.82%) ⬆️
pkg_settings_library 90.19% <ø> (ø)
pkg_simcore_sdk 84.97% <ø> (-0.06%) ⬇️
agent 93.53% <ø> (ø)
api_server 91.95% <95.65%> (+0.01%) ⬆️
autoscaling 95.77% <ø> (ø)
catalog 92.36% <ø> (ø)
clusters_keeper 99.13% <ø> (ø)
dask_sidecar 91.81% <ø> (ø)
datcore_adapter 97.94% <ø> (ø)
director 75.81% <ø> (ø)
director_v2 90.97% <ø> (-0.03%) ⬇️
dynamic_scheduler 96.27% <ø> (ø)
dynamic_sidecar 90.46% <ø> (ø)
efs_guardian 89.62% <ø> (ø)
invitations 91.44% <ø> (ø)
payments 92.61% <ø> (ø)
resource_usage_tracker 92.24% <ø> (ø)
storage 86.45% <100.00%> (-0.29%) ⬇️
webclient ∅ <ø> (∅)
webserver 87.97% <100.00%> (-0.03%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae14233...47ab209. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mergify
Copy link
Contributor

mergify bot commented Sep 11, 2025 •
edited
Loading

🧪 CI Insights

Here's what we observed from your CI run for 47ab209.

✅ Passed Jobs With Interesting Signals

Pipeline Job Signal Health on master Retries 🔍 CI Insights 📄 Logs
CI system-tests Base branch is broken, but the job passed. Looks like this might be a real fix 💪 Broken 0 View View

@bisgaard-itis bisgaard-itis marked this pull request as ready for review September 11, 2025 13:12
sanderegg
sanderegg approved these changes Sep 11, 2025
View reviewed changes
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very nice! thanks!

GitHK
GitHK reviewed Sep 12, 2025
View reviewed changes
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, just a questions

pcrespov
pcrespov approved these changes Sep 12, 2025
View reviewed changes
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

@bisgaard-itis bisgaard-itis requested a review from GitHK September 12, 2025 10:20
@sonarqubecloud
Copy link

sonarqubecloud bot commented Sep 12, 2025

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@bisgaard-itis bisgaard-itis enabled auto-merge (squash) September 12, 2025 11:43
@bisgaard-itis
Copy link
Contributor Author

bisgaard-itis commented Sep 12, 2025

@Mergifyio queue

@mergify
Copy link
Contributor

mergify bot commented Sep 12, 2025 •
edited
Loading

queue

🟠 Waiting for conditions to match

  • -closed [📌 queue requirement]
  • -conflict [📌 queue requirement]
  • -draft [📌 queue requirement]
  • any of: [📌 queue -> configuration change requirements]
    • -mergify-configuration-changed
    • check-success = Configuration changed
  • any of: [🔀 queue conditions]
    • all of: [📌 queue conditions of queue default]
      • #approved-reviews-by >= 2 [🛡 GitHub branch protection]
      • #approved-reviews-by>=2
      • #changes-requested-reviews-by = 0 [🛡 GitHub branch protection]
      • #changes-requested-reviews-by=0
      • #review-threads-unresolved = 0 [🛡 GitHub branch protection]
      • #review-threads-unresolved=0
      • -conflict
      • -draft
      • base=master
      • branch-protection-review-decision = APPROVED [🛡 GitHub branch protection]
      • label!=🤖-do-not-merge
      • label=🤖-automerge
      • any of: [🛡 GitHub branch protection]
        • check-skipped = deploy to dockerhub
        • check-neutral = deploy to dockerhub
        • check-success = deploy to dockerhub
      • any of: [🛡 GitHub branch protection]
        • check-success = system-tests
        • check-neutral = system-tests
        • check-skipped = system-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = unit-tests
        • check-neutral = unit-tests
        • check-skipped = unit-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = check OAS' are up to date
        • check-neutral = check OAS' are up to date
        • check-skipped = check OAS' are up to date
      • any of: [🛡 GitHub branch protection]
        • check-success = integration-tests
        • check-neutral = integration-tests
        • check-skipped = integration-tests
      • any of: [🛡 GitHub branch protection]
        • check-success = build-test-images (frontend) / build-test-images
        • check-neutral = build-test-images (frontend) / build-test-images
        • check-skipped = build-test-images (frontend) / build-test-images
      • any of: [🛡 GitHub branch protection]
        • check-success = SonarCloud Code Analysis
        • check-neutral = SonarCloud Code Analysis
        • check-skipped = SonarCloud Code Analysis

@bisgaard-itis bisgaard-itis added the 🤖-automerge marks PR as ready to be merged for Mergify label Sep 12, 2025
@bisgaard-itis bisgaard-itis merged commit 381dc67 into ITISFoundation:master Sep 12, 2025
94 of 95 checks passed
This was referenced Sep 12, 2025
Merged
Closed
This was referenced Sep 18, 2025
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@giancarloromeo giancarloromeo giancarloromeo approved these changes

@pcrespov pcrespov pcrespov approved these changes

@sanderegg sanderegg sanderegg approved these changes

@matusdrobuliak66 matusdrobuliak66 Awaiting requested review from matusdrobuliak66 matusdrobuliak66 is a code owner

@GitHK GitHK Awaiting requested review from GitHK GitHK is a code owner

Assignees

@bisgaard-itis bisgaard-itis

Labels

🤖-automerge marks PR as ready to be merged for Mergify a:celery-library

Projects

None yet

Milestone

Cheops

Development

Successfully merging this pull request may close these issues.

[Bug]: Celery task filtering shows incorrect tasks Evolve TaskContext data structure to a TaskFilter

5 participants

@bisgaard-itis @giancarloromeo @GitHK @pcrespov @sanderegg